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1.  Introduction 


1.1  Purpose 

The  goal  of  this  research  was  to  examine  if  gunners  in  a  Future  Combat  System  (FCS)  vehicle 
such  as  the  Mounted  Combat  System  (MCS)  were  able  to  effectively  maintain  local  security  (i.e., 
perform  their  gunners’  tasks)  while  managing  their  unmanned  assets.  According  to  Mitchell 
(2005),  which  examined  workload  for  MCS  crew  members  using  human  performance  modeling, 
the  gunner  is  the  most  viable  option  for  controlling  robotic  assets,  compared  to  the  other  two 
positions  (i.e.,  commander  and  driver).  This  current  research  examined  if  the  gunner  could 
effectively  detect  targets  in  his  or  her  immediate  environment  while  operating  robotic  assets  in  a 
remote  environment. 

1.2  Background 

Past  research  in  dual  task  performance  suggests  that  operators  may  encounter  difficulties  when 
both  tasks  involve  focal  vision  (Horrey  &  Wickens,  2004).  Horrey  and  Wickens  (2004) 
demonstrated  that  participants  could  not  effectively  detect  road  hazards  while  operating  in- 
vehicle  devices.  Additionally,  research  about  visual  performance  demonstrated  that  as  the  size 
of  the  search  set  increased,  performance  degraded  in  terms  of  speed,  accuracy,  or  both  (Scanlan, 
1977).  Murray  (1994)  showed  that  as  the  number  of  monitored  displays  increased,  operators’ 
reaction  time  for  their  target  search  tasks  also  increased  linearly.  In  fact,  reaction  time  almost 
doubled  when  the  number  of  displays  increased  from  1  to  2  and  from  2  to  3  (a  slope  of  1 .94  was 
obtained). 

According  to  Wickens,  Dixon,  and  Chang  (2003),  visual  angle  separation  larger  than  about  6.4  to 
7.5  degrees  may  degrade  event  monitoring  response  time.  In  the  case  of  concurrent  performance 
of  a  gunner’s  and  robotic  operator’s  tasks,  performance  was  expected  to  be  worse  than  when  the 
operator  only  had  to  perfonn  one  task  since  concurrent  tasks  involved  more  displays  to  visually 
scan.  The  gunner’s  task  performance  was  expected  to  further  degrade  when  the  robotic  tasks 
became  more  challenging  (i.e.,  when  more  than  mere  monitoring  was  needed),  for  example,  when 
robots  needed  teleoperation  and/or  when  the  operator  needed  to  use  the  user  interface  to  perform 
some  tasks  (e.g.,  putting  targets  on  the  map,  labeling  the  targets,  sending  spot  reports,  etc.). 

In  addition,  research  has  shown  that  increased  mental  workload  could  reduce  the  size  of 
operator’s  visual  field  (Rantanen  &  Goldberg,  1999).  It  was  expected  that  the  reduced  visual 
field  would  have  a  significant  impact  on  the  operator’s  gunnery  task  performance  (i.e.,  target 
detection  in  his  immediate  environment). 
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Finally,  signal-to-noise  ratios  (SNRs)  were  expected  to  impact  an  operator’s  performance 
(Wickens,  1992).  As  visual  noise  increased  in  the  gunner’s  immediate  environment  or  in  the 
remote  environment  where  the  robots  were  located,  the  gunner’s  target  detection  performance 
was  expected  to  degrade. 

Participants’  robotic  task  perfonnance  is  expected  to  differ,  depending  on  the  type  of  asset 
available  and  the  type  of  task  they  are  asked  to  perform.  Chen,  Durlach,  Sloan,  and  Bowens 
(2005)  demonstrated  that  participants’  target  detection  was  significantly  lower  when  they  had  to 
teleoperate  the  unmanned  ground  vehicle  (UGV)  as  compared  to  when  the  UGV  was  semi- 
autonomous.  Chen  et  al.  suggested  that  maybe  participants’  teleoperation  (i.e.,  driving  the  robot) 
negatively  affected  their  target  detection  performance.  Luck,  Allender,  and  Russell  (2006) 
reported  that  robotic  operators’  situational  awareness  (SA)  was  better  when  the  small  UGV  had  a 
higher  level  of  automation.  Luck  et  al.  suggested  that  the  attention  on  (manual)  robotic  control 
might  have  distracted  the  operators  from  focusing  on  the  vehicle’s  location,  which  was  the 
study’s  measure  of  SA.  Dixon,  Wickens,  and  Chang  (2003)  also  reported  that  pilots  found  more 
targets  when  their  unmanned  aerial  vehicle(s)  were  autonomous  than  when  they  were 
teleoperated. 

1.3  Current  Study 

In  this  study,  we  simulated  an  MCS  environment  and  conducted  an  experiment  to  examine  the 
workload  and  performance  of  the  combined  position  of  gunner  and  robotic  operator.  In  one 
condition  (Gunner  Baseline),  participants  had  to  perform  the  gunnery  tasks  only  (i.e.,  target 
detection  and  engagement);  in  the  other  three  conditions,  they  had  to  monitor  or  manage  a  UGV 
via  the  robotic  operator  control  unit  (OCU)  while  simultaneously  performing  their  gunner’s 
tasks.  In  one  of  the  three  concurrent  task  conditions  (Monitor  condition),  participants  only  had 
to  monitor  the  UGV  via  the  video  feed  for  targets;  in  the  second  condition  (UGV  condition),  they 
had  to  actively  manage  the  semi-autonomous  UGV  as  well  as  monitor  its  video  feed;  in  the  third 
condition  (Teleop  condition),  they  had  to  teleoperate  the  UGV  and  monitor  its  video  feed. 

Visual  density  levels  (i.e.,  density  of  targets)  were  manipulated  in  the  remote  environment  of  the 
UGV  and  Teleop  conditions  so  their  effects  on  gunner’s  perfonnance  could  be  examined. 

Participants  also  had  to  concurrently  perform  a  tertiary  communication  task,  which  simulated 
gunner’s  communication  with  fellow  crew  members  in  the  vehicle.  Richard,  Wright,  Ee,  Prime, 
Shimizu,  and  Vavrik  (2002)  found  that  participants’  change  detection  was  negatively  affected  by 
a  concurrent  auditory  task.  More  specifically,  participants’  reaction  times  were  slower  and 
visual  scanning  was  less  effective.  In  the  current  study,  we  expected  the  concurrent 
communication  to  have  a  similar  negative  effect  on  participants’  target  detection  perfonnance. 
Although  we  did  not  manipulate  the  communication  task  as  a  variable,  we  tried  to  examine  if 
participants  with  higher  attentional  control  could  perfonn  their  tasks  more  effectively  than  those 
with  lower  attentional  control  in  our  simulated  multi-tasking  environment.  Schumacher  et  al. 
(2001)  demonstrated  that  some  participants  were  more  effective  in  concurrently  performing  a 
visual  task  and  an  auditory  task  but  did  not  examine  what  individual  difference  factor(s) 
contributed  to  that  time-sharing  effectiveness. 
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Finally,  the  relationship  between  participants’  spatial  ability  (SpA)  and  their  task  performance 
was  examined.  According  to  Chen  et  al.  (2005),  those  with  higher  SpA  perfonned  target 
detection  tasks  using  robotic  assets  more  effectively  than  those  with  lower  SpA.  In  the  current 
study,  two  different  types  of  spatial  tests  were  employed.  It  was  expected  that  those  with  higher 
SpA  test  scores  would  perform  their  robotic  tasks  better. 


2.  Method 


2.1  Participants 

A  total  of  20  students  (3  females  and  17  males)  was  recruited  from  the  University  of  Central 
Florida  and  participated  in  the  study.  The  ages  of  the  participants  ranged  from  18  to  45  (M=  20.8, 
SD  =  3.2).  Participants  were  compensated  $40  and  were  given  class  credit  for  their  participation  in 
the  experiment. 

2.2  Apparatus 
2.2.1  Simulators 

The  experiment  was  conducted  with  a  tactical  control  unit  (TCU)  developed  by  the  U.S.  Army 
Research  Laboratory’s  (ARL’s)  Robotics  Collaborative  Technology  Alliance  (RCTA)  for  the 
robotic  control  tasks.  The  gunnery  component  was  implemented  with  an  additional  screen  and 
controls  to  simulate  the  out-the-window  (OTW)  view  and  line-of-sight  (LOS)  and  beyond-line- 
of-sight  (BLOS)  firing  capabilities. 

The  RCTA  TCU  is  a  one-person  crew  station  from  which  the  operator  can  control  several 
simulated  robotic  assets,  which  can  perform  tasks  semi-autonomously  or  be  teleoperated  (see 
figure  1).  The  operator  switched  operation  modes  and  display  modes  through  the  use  of  a  19- 
inch  touch-screen  display.  A  joystick  was  used  to  manipulate  the  direction  in  which  the 
unmanned  vehicles  moved  when  in  Teleop  mode.  The  UGV  simulated  in  our  study  is  the 
experimental  unmanned  vehicle  (XUV)  developed  by  ARL.  The  simulation  program  used  in 
this  study  was  rSAF,  which  is  a  version  of  OneSAF  (one  semi-automated  force)  for  robotics 
simulation. 

The  gunnery  component  consisted  of  a  monitor  and  a  joystick  (see  figure  2).  The  interface 
consists  of  a  15 -inch  flat  panel  monitor  and  a  joystick.  Participants  used  the  joystick  to  rotate  the 
sensors  360  degrees,  zoom  in  and  out,  switch  between  firing  modes,  and  engage  targets.  For 
engaging  BLOS  targets,  the  participants  need  to  receive  authorization  from  the  vehicle 
commander  (i.e.,  the  experimenter),  align  their  aim  with  the  direction  of  the  target  (the  line 
would  turn  red  when  it  was  aiming  at  the  target),  and  then  fire. 


3 


o 


Drag 

Mosaic 


<K 


Grab 

Mosaic 


Grab 

Snapshot 


Invert 

IR 


Add 
T  arget 


Laze 


FOV 


lozoooooil 


I02000002 


View 

Details 


Asset 


Acquire 


Track: 


Zoom: 

100% 


Teleop 


Show 

All 


Clear 


Pause 


Orders 


XUV1 

6256E  Done 

7902 N  2.0  kph  0" 

I  XUV2 

6276E  Teleop 

7912 N  5.0  kph  1" 

XUV3 

6296E  Convoy  LAV2  50m 

7922 N  9.0  kph  1' 

XUV4 

6316E  Convoy  CHold)  LAV2  5C 
7932 N  14.0  kph  2' 

m 

XUVS 

6336E  Stuck 

7942 N  20.0  kph  2" 

XUV6 

6356E  Inoperable 

7952 N  27.0  kph  3' 

OCU  with  RSTA  view 


Figure  1.  User  interface  of  RCTA  TCU. 
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Cognitive  tests  were  administered  concurrently  with  the  experimental  sessions.  The  questions 
included  simple  military-related  reasoning  tests  and  simple  memory  tests.  The  inclusion  of  these 
cognitive  tasks  was  for  simulating  an  environment  where  the  gunner  was  communicating  with 
fellow  crew  members  in  the  vehicle.  For  the  reasoning  tests,  there  were  questions  such  as  “if  the 
enemy  is  to  our  left,  and  our  UGV  is  to  our  right,  what  direction  is  the  enemy  to  the  UGV?”  For 
the  memory  tests,  the  participants  were  asked  to  repeat  some  short  statements  or  keep  track  of 
three  radio  call  signs  (e.g.,  Bravo  83)  and  they  had  to  report  to  the  experimenter  whether  the  call 
signs  they  heard  were  one  of  those  they  were  tracking.  Test  questions  were  delivered  by  a 
synthetic  speech  program,  DECTalk1. 

The  questions  were  pre-recorded  by  a  male  speaker  and  presented  at  the  rate  of  one  question 
approximately  every  33  seconds. 

2.2.2  Questionnaires  and  Tests 

A  demographics  questionnaire  was  administered  at  the  beginning  of  the  training  session 
(appendix  A). 

A  questionnaire  about  attentional  control  (appendix  B)  (Derryberry  &  Reed,  2002)  was  used  to 
evaluate  participants’  perceived  attentional  control  (PAC).  The  attentional  control  survey 
consists  of  21  items  and  measures  attention  focus  and  shifting.  The  Cube  Comparison  Test 
(CCT)  (Educational  Testing  Service,  2005)  and  the  Spatial  Orientation  Test  (SOT)  were  used  to 
assess  participants’  SpA.  The  CCT  required  participants  to  compare  in  3  minutes,  21  pairs  of 
six-sided  cubes  and  determine  if  the  rotated  cubes  were  the  same  or  different.  The  SOT, 
constructed  by  Dr.  Paula  Durlach  of  the  U.S.  Army  Research  Institute,  is  modeled  after  the 
cardinal  direction  test  developed  by  Gugerty  and  his  colleagues  (Gugerty  &  Brooks,  2004)  and  is 
a  computerized  test  consisting  of  a  brief  training  segment  and  32  test  questions.  Both  accuracy 
and  response  time  were  automatically  captured  by  the  program.  Participants’  perceived 
workload  was  evaluated  via  the  National  Aeronautics  and  Space  Administration  task  load  index 
(NASA-TLX)  questionnaire  (appendix  C)  (Hart  &  Staveland,  1988).  The  NASA-TLX  is  a  self- 
reported  questionnaire  of  perceived  demands  in  nine  areas:  mental,  physical,  temporal,  effort 
(mental  and  physical),  frustration,  performance,  visual,  cognitive,  and  psychomotor.  Participants 
were  asked  to  evaluate  their  perceived  workload  level  in  these  areas  on  10-point  scales. 

The  Simulator  Sickness  Questionnaire  (SSQ)  (see  appendix  D)  was  used  to  evaluate  participants’ 
simulator  sickness  symptoms  (Kennedy,  Lane,  Berbaum,  &  Lilienthal,  1993).  The  SSQ  consists 
of  a  checklist  of  16  symptoms.  Each  symptom  is  related  in  terms  of  degrees  of  severity  (none, 
slight,  moderate,  severe).  A  total  severity  (TS)  score  can  be  derived  by  a  weighted  scoring 
procedure  and  reflects  overall  discomfort  level. 


1  DECtalk  is  a  registered  trademark  of  Digital  Equipment  Corporation. 
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Finally,  a  usability  questionnaire  was  constructed  (see  appendix  E),  based  on  the  one  used  in  the 
Unmanned  Combat  Demonstration  (UCD)  study,  since  the  test  bed  used  in  our  study  was 
modeled  after  the  crew  station  investigated  in  the  UCD  study  (Kamsickas,  2003).  Specifically, 
the  questionnaire  included  the  following  sections:  asset  summary;  reconnaissance,  surveillance, 
and  target  acquisition  (RSTA);  map  display;  teleoperation;  reporting,  and  general  usability  of  the 
TCU.  Participants  indicated  their  level  of  agreement  with  the  items  using  7-point  numerical 
scales.  Participants  were  also  given  an  opportunity  to  provide  comments  to  support  or  clarify 
their  numerical  responses.  The  comments,  in  addition  to  the  numerical  responses,  provided  the 
researchers  with  further  insight  as  to  the  participants’  opinions  about  the  crew  station. 

2.3  Experimental  Design 

The  overall  design  of  the  study  is  a  repeated  measures  design.  There  were  six  conditions: 

•  Gunnery  Baseline  (Gunner  Baseline) 

•  Concurrent  task  conditions: 

o  Monitor:  Gunnery  +  Monitoring  1  Semi-autonomous  UGV  (Monitor) 
o  Gunnery  +  Monitoring  +  Control  of  1  Semi-autonomous  UGV 

■  Low  Density  (UGV-Low) 

■  High  Density  (UGV-High) 

o  Gunnery  +  Monitoring  +  Teleoperating  UGV 

■  Low  Density  (Teleop-Low) 

■  High  Density  (Teleop-High) 

Target  density  levels  were  manipulated  in  the  UGV  and  Teleop  conditions.  In  these  conditions, 
ratios  of  target  versus  noise  (i.e.,  neutral  entities  such  as  civilians  and  civilian  vehicles)  were 
manipulated.  In  the  high  density  areas,  the  SNR  was  1:3;  in  the  low  density  areas,  the  ratio  was 
1:1. 

For  the  purpose  of  comparing  participants’  robotic  control  task  perfonnance  between  the  single¬ 
task  and  concurrent-task  conditions,  we  added  a  UGV  Baseline  condition  to  half  of  the 
participants  and  a  Teleop  Baseline  to  the  other  half  of  the  participants.  In  these  two  conditions, 
they  did  not  have  to  perfonn  the  gunnery  tasks  and  only  had  to  simultaneously  perfonn  the 
robotic  control  tasks  and  the  communication  tasks. 

2.4  Procedure 

After  being  briefed  about  the  purpose  of  the  study,  the  tasks  for  the  experiment,  and  any  risks 
involved,  participants  read  and  signed  a  consent  form.  They  then  answered  the  attentional 
control  survey  and  were  administered  the  SpA  tests  (i.e.,  the  CCT  and  the  SOT).  After  these 
tests,  participants  received  training,  which  lasted  approximately  2  hours.  Training  was  self- 
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paced  and  was  delivered  by  PowerPoint2  slides  showing  the  elements  of  the  TCU,  steps  for 
completing  various  tasks,  several  mini-exercises  for  practicing  the  steps,  and  two  exercises  for 
performing  the  robotic  control  tasks  (one  for  practicing  the  Teleoperation  task  and  one  for 
practice  of  the  UGV  control  tasks).  After  the  tutorial  on  TCU,  participants  were  trained  in  the 
gunnery  tasks  and  completed  an  exercise  including  LOS  and  BLOS  firing  procedures.  After  the 
participants  were  familiar  with  the  gunnery  tasks,  they  completed  one  final  exercise  in  which 
they  performed  the  gunnery  tasks  and  the  robotic  control  tasks  at  the  same  time.  At  this  point,  all 
tutorial  materials  and  information  were  removed  and  the  participants  had  to  be  able  to  perfonn 
all  these  tasks  on  their  own.  After  this  final  exercise,  the  experimenter  detennined  if  the 
participant  needed  any  further  practice  on  the  robotic  control  tasks  or  gunnery  tasks  and  provided 
some  further  training  and  exercises  if  necessary.  Participants  had  to  demonstrate  that  they  could 
recall  all  the  steps  for  perfonning  the  tasks  without  any  help. 

The  experimental  session  took  place  on  a  different  day  but  within  a  week  of  the  training  session. 
Participants’  tasks  were  to  use  their  robotic  assets  to  locate  targets  in  the  remote  environment 
(mixture  of  enemy  tanks  and  dismounted  Soldiers)  and  targets  in  their  immediate  (i.e.,  MCS) 
environment.  There  were  10  targets  in  each  environment.  The  MCS  was  simulated  as  traveling 
along  a  designated  route.  They  did  this  six  times  (i.e.,  six  experimental  sessions,  each  lasting 
approximately  15  minutes),  and  the  order  of  experimental  conditions  was  counterbalanced  across 
participants. 

For  the  Gunner  Baseline,  the  operator  performed  only  gunnery  tasks  (i.e.,  target  detection  and 
engagement).  In  the  remaining  concurrent  task  conditions,  participants  monitored  the  screens 
which  simulated  the  OTW  views  and  engaged  targets  as  they  were  detected  while  performing  the 
robotic  tasks  at  the  same  time.  Some  of  the  targets  in  the  immediate  environment  were  BLOS. 
Participants  needed  to  conduct  BLOS  fires  when  those  targets  were  detected  by  the  robotic 
assets.  There  were  three  concurrent  task  conditions:  Monitor,  UGV,  and  Teleop.  The  Monitor 
condition  required  the  operator  to  monitor  the  video  feed  as  the  UGV  traveled  and  report  any 
targets  detected  (note:  only  human  targets  were  used  in  the  Monitor  condition).  The  UGV 
conditions  required  the  operator  to  monitor  the  video  feed  as  the  UGV  traveled,  examine  still 
images  generated  from  the  reconnaissance  scans  (i.e.,  RSTA  scans),  which  were  enabled  by  the 
aided  target  recognition  (ATR)  capabilities,  and  detect  targets.  The  Teleop  conditions  required 
the  operator  to  manually  manipulate  and  drive  the  UGV  along  a  predetermined  route  using  the 
TCU  to  detect  targets.  In  the  UGV  and  Teleop  conditions,  upon  detecting  a  target,  participants 
needed  to  place  the  target  on  the  map,  label  the  target,  and  then  send  a  spot  report.  A  list  of 
robotic  tasks  for  the  UGV  and  Teleop  conditions  is  presented  in  table  1. 


9  .  . 

“PowerPoint  is  a  trademark  of  Microsoft  Corporation. 
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Table  1.  Robotic  tasks  for  the  UGV  and  teleop  conditions. 


Condition 

UGV 

Teleop 

Tasks 

Identify  target  or  neutral 

Identify  target  or  neutral 

Verbally  report  neutrals 

Verbally  report  neutrals 

Queue  target  (i.e.,  add  to  map) 

Switch  to  map  display 

Switch  to  map  display 

Add  target  to  the  map 

Label  target 

Label  target 

Submit  spot  report 

Submit  spot  report 

There  were  civilians  and  civilian  vehicles  in  the  simulated  environment  to  increase  the  visual 
noise  for  the  target  detection  tasks.  In  the  remote  environment,  there  were  high  density  areas  and 
low  density  areas.  In  the  high  density  areas,  the  SNR  was  1 :3;  in  the  low  density  areas,  the  ratio 
was  1:1.  In  the  immediate  environment,  the  SNR  was  constant  throughout  the  route. 

While  the  participants  were  perfonning  their  gunnery  and/or  robotic  control  tasks,  they  had  to 
perform  the  communication  tasks  by  answering  questions  delivered  to  them  via  a  synthetic 
speech  program,  DECtalk. 

Half  of  the  participants  also  completed  the  UGV  Baseline  condition  and  the  other  half  Teleop 
Baseline  condition,  in  which  they  did  not  have  to  perform  the  gunnery  tasks  concurrently.  Their 
only  tasks  were  robotic  and  communication  tasks.  The  UGV  Baseline  and  Teleop  Baseline 
conditions  were  completed  right  after  their  respective  concurrent  task  conditions  (i.e.,  UGV  and 
Teleop).  These  two  robotic  baseline  conditions  allowed  us  to  examine  perfonnance  degradation 
in  robotic  tasks  associated  with  concurrent  task  conditions  (i.e.,  when  the  operator  had  to 
perform  both  the  gunnery  and  the  robotic  tasks).  There  were  2-minute  breaks  between 
experimental  sessions. 

Participants  assessed  their  workload  using  the  NASA-TLX  after  they  completed  each 
experimental  session,  except  for  the  UGV  and  Teleop  conditions.  They  only  answered  the 
questionnaire  after  they  completed  both  the  high  and  low  visual  density  conditions.  At  the 
conclusion  of  all  scenarios,  participants  were  administered  the  SSQ,  used  to  evaluate  the  severity 
of  their  simulator  sickness  symptoms.  The  participants  also  completed  a  usability  questionnaire 
regarding  the  TCU  at  the  end  of  the  experimental  session. 

2.5  Measures 

The  dependent  measures  include  mission  performance  (i.e.,  number  of  targets  detected  in  the 
remote  environment  using  the  robotic  assets  and  number  of  enemy  targets  detected  in  the 
immediate  environment),  communication  task  perfonnance,  and  perceived  workload. 
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3.  Results 


3.1  Target  Detection  Performance 


3.1.1  Gunnery  Tasks 

Table  2  lists  several  measures  relating  to  enemy  target  detection.  Correlations  between  partici¬ 
pants’  gunnery  task  performance  and  their  attentional  control,  cube  test,  and  SOT  scores  were  first 
evaluated.  The  SOT  scores  were  found  to  be  the  most  accurate  predictor  of  participants’  gunnery 
performance,  with  Gunner  Baseline,  Monitor,  and  Teleop  conditions  being  significant  (r’s  =  0.408, 
0.506,  0.55,  all  p's  <  0.05).  Participants  were  then  designated  as  high  spatial  ability  or  low  spatial 
ability,  based  on  their  SOT  scores  (median  split).  A  mixed  analysis  of  variance  (ANOVA)  was 
performed  to  examine  the  effects  of  the  concurrent  robotic  control  tasks  on  the  gunnery  task 
performance,  with  the  Asset  condition  (Gunner  Baseline,  Monitor,  UGV-Low,  UGV-High, 
Teleop-Low,  and  Teleop-High)  being  the  within-subject  factor  and  SpA  (SOT  score)  as  the 
between-subject  factor.  The  analysis  revealed  that  the  Asset  condition  significantly  affected 
number  of  targets  detected,  F( 5,  15)  =  132,  p  <  0.001,  with  Gunner  Baseline  being  the  highest  and 
Teleop-High  being  the  lowest  (figure  3).  Post  hoc  tests  (Least  Significant  Difference  or  LSD) 
showed  that  target  detection  in  Gunner  Baseline  was  significantly  higher  than  every  condition 
except  Monitor.  Monitor  was  significantly  higher  than  all  the  UGV  and  Teleop  conditions. 
Teleop-High  was  significantly  lower  than  both  UGV  conditions.  Participants  with  higher  SpA  had 
significantly  higher  gunnery  task  performance  than  did  those  with  lower  SpA,  F(  1,  18)  =  8.76, 
p  <  0.005  (figure  4). 


Table  2.  Mean  proportion  of  targets  detected  (standard  deviations  are  shown  in  parentheses). 


Condition 

Measures 

Gunner 

Baseline 

Monitor 

UGV- 

Low 

UGV- 

High 

Teleop- 

Low 

Teleop- 

High 

Gunnery  Task 
(Enemy  targets  detected) 

.69a 

(.174) 

.645“ 

(.196) 

250? 

(.191) 

.495b 

(.176) 

A8? 

(.228) 

.405c 

(.161) 

Robotic  Task 
(human  targets  only) 

NA 

.6300“ 

(.138) 

.5983b 

(.1441) 

,4653c 

(.2063) 

.7052“ 

(.2167) 

.5754“ 

(.2243) 

Robotic  Task 
(vehicle  targets  only) 

NA 

NA 

.8917“ 

(.1271) 

.8506“ 

(.1329) 

.9167“ 

(.1239) 

.8822“ 

(.1433) 

“Note:  Statistics  with  the  same  superscript  are  not  significantly  different  from  one  another 
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3.1.2  Robotic  Tasks 


None  of  the  individual  difference  factors  (i.e.,  PAC  and  SpA  measured  by  the  SOT  and  the  CCT) 
were  found  to  consistently  correlate  with  participants’  robotic  task  performance.  SOT  scores 
correlated  significantly  with  robotic  task  performance  in  the  UGV  condition,  r  =  .489  (p  =  0.014) 
but  not  in  other  conditions.  We  also  examined  participants’  speed  in  performing  the  SOT  and 
found  that  it  correlated  negatively  with  robotic  task  performance  in  the  Teleop  condition, 
r  =  -0.463  (p  =  0.02). 


Two  repeated  measures  ANOVAs  were  perfonned  to  compare  participants’  target  detection 
performance  on  the  robotic  tasks,  one  for  the  human  targets  and  the  other  for  the  vehicle  targets. 
The  UGV-Low  and  UGV-High  conditions  were  combined  as  the  UGV  condition,  and  the  Teleop- 
Low  and  Teleop-High  conditions  were  combined  as  the  Teleop  condition.  The  first  analysis 
showed  there  were  significant  differences  among  the  Monitor,  UGV,  and  Teleop  conditions  in 
human  target  detection,  F( 2,  18)  =  4.794 ,p  <  0.05,  with  UGV  being  the  lowest  (see  figure  5).  Post 
hoc  tests  (LSD)  showed  that  differences  between  Monitor  and  UGV  and  between  Teleop  and  UGV 
were  significant.  The  second  analysis  examined  the  difference  between  the  UGV  and  the  Teleop 
conditions  in  vehicle  target  detection  and  the  difference  was  not  significant. 


Robotic  Task  Performance 


Robotic  Task 


Error  Bars  show  95.0%  Cl  of  Mean 

Bars  show  Means 


Figure  5.  Robotic  task  performance. 


To  further  examine  why  the  Teleop  condition  produced  better  target  detection  rates  than  the 
UGV  condition,  which  conflicted  with  the  findings  in  Chen  et  al.  (2005),  we  compared  the 
amounts  of  targets  detected  along  the  route  and  targets  detected  within  RSTA  areas  for  the  UGV 
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and  Teleop  conditions.  A  2  x  2  repeated  measures  ANOVA  was  perfonned  with  Asset  condition 
(UGV  versus  Teleop)  and  Target  Location  (Route  versus  RSTA)  as  the  factors.  The  analysis 
revealed  that  both  effects  were  significant:  Asset,  F{  1,  38)  =  5.75,  p  =  0.019;  Location  F(  1,  38) 

=  18.01,/?  <  0.0001.  Post  hoc  tests  showed  that  the  largest  difference  in  targets  detected  was 
along  the  route,  with  UGV  having  a  35%  and  Teleop  51%  target  detection  rate. 

Half  of  the  participants  also  completed  the  UGV  Baseline  condition  and  the  other  half  Teleop 
Baseline  condition  so  we  could  compare  if  participants’  target  detection  performance  degraded 
when  they  had  to  perfonn  the  gunnery  task  concurrently.  Results  showed  that  when  participants 
only  had  to  operate  the  UGV,  their  overall  target  detection  rate  (including  both  human  and 
vehicle  targets)  was  80%;  when  they  had  to  concurrently  operate  the  UGV  while  performing  the 
gunner’s  tasks  (i.e.,  UGV  condition),  their  target  detection  using  the  UGV  dropped  to  67% 
(difference  marginally  significant).  These  results  are  presented  graphically  in  figure  6. 

The  effect  of  visual  density  was  examined  with  a  2  x  2  repeated  measures  ANOVA  with  the  Asset 
(UGV  versus  Teleop)  and  Visual  Density  (High  versus  Low)  as  the  factors.  The  effect  of  visual 
density  was  significant,  F(  \ ,  38)  =  7.875,/?  =  0.008,  with  lower  target  detection  performance 
associated  with  higher  visual  density  (figure  7).  None  of  the  individual  difference  factors  was 
significant. 

Robotic  Task  Performance 

Baseline  vs.  Concurrent  Conditions 


Baseline  vs.  Concurrent 

| ® —  Baseline 
H  Concurrent 

Error  Bars  show  95.0%  Cl  of  Mean 

Bars  show  Means 


UGV  Teleop 

_ Robotic  Task _ 

Figure  6.  Comparisons  between  robotic-baseline  and  robotic-concurrent. 
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Robotic  Task  Performance 

Effects  of  Visual  Density 


UGV  Teleop 
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Visual  Density 

■ - * -  1.00  Low 

H  -*>-  2.00  High 


Robotic  Task 


Figure  7.  Effects  of  visual  density  on  robotic  task  performance. 


3.2  Communication  Task  Performance 

The  differences  among  the  four  conditions  were  significant,  F(3,  16)  =  6.514,  p  <  0.005,  with  the 
Gunner  Baseline  and  Monitor  conditions  being  higher  than  the  UGV  and  Teleop  conditions. 
Participants’  PAC  was  identified  by  their  attentional  control  survey  scores.  Attentional  control 
scores  were  found  to  be  positively  correlated  with  the  communication  task  performance  in  the 
Teleop  conditions,  r=  0.476  ip  =  0.023).  For  the  UGV  condition,  the  correlation  was  marginally 
significant  r=  0.385  ip  =  0.057).  Results  are  graphically  presented  in  figure  8. 
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Communication  Task  Performance 

Effects  of  Perceived  Attentional  Control 
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Figure  8.  Communication  task  performance. 


3.3  Perceived  Workload 

Unweighted  ratings  of  the  scales  of  the  NASA-TLX  were  used  for  this  analysis.  Participants’ 
self-assessment  of  workload  was  significantly  affected  by  Asset  condition,  F( 3,  17)  =  65.102, 
p  <  0.0001.  The  perceived  workload  was  highest  in  the  Teleop  condition  (M=  43.025)  and 
lowest  in  the  Gunner  Baseline  condition  (M=  22.35).  All  pairwise  comparisons  were 
significant.  Results  are  presented  graphically  in  figure  9.  Correlations  between  participants’ 
perceived  workload  and  their  PAC,  CCT,  and  SOT  scores  were  evaluated.  The  correlations 
between  PAC  and  workload  were  all  negative  (those  with  higher  PAC  had  lower  perceived 
workload  than  did  those  with  lower  scores),  but  only  Teleop  reached  significance,  r=  -0.5 16 
{p  =  0.012).  The  correlations  between  CCT  scores  and  workload  were  all  positive  (those  with 
higher  CCT  scores  had  higher  perceived  workload  than  did  those  with  lower  scores),  with  UGV 
and  Teleop  reached  significance,  r=  0.587  (p  =  0.003)  and  r=  0.484  (p  =  0.015),  respectively. 
The  correlations  between  SOT  scores  and  workload  were  less  consistent,  with  only  UGV  reached 
significance,  r=  0.441  (p  =  0.026).  Correlations  between  participants’  reaction  times  for  the 
SOT  and  their  perceived  workload  were  also  evaluated.  The  correlations  were  also  negative 
(those  who  were  faster  had  higher  perceived  workload  than  did  those  with  lower  scores),  but 
none  reached  significance. 


14 


3.4  Simulator  Sickness 

Participants’  simulator  sickness  scores  (SSQ:  three  sub-scales  and  the  total  severity  score  [TSS]) 
were  calculated  on  the  basis  of  formulae  in  Kennedy  et  al.  (1993)  (see  appendix  F  for  the  scoring 
procedure).  The  average  TSS  was  29.36  ( SD  =  24.06).  Further  examination  of  the  sub-scale 
data  indicated  that  the  oculomotor  aspect  significantly  contributed  to  the  elevated  TSS.  In  order 
to  evaluate  the  relationship  between  simulator  sickness  and  performance  and  perceived 
workload,  we  calculated  aggregated  workload  ratings,  gunnery  performance,  and  robotic  task 
performance  by  averaging  the  data  across  conditions.  We  found  that  there  was  a  significant 
correlation  between  TSS  and  workload,  r  =  0.379  (p  =  0.05).  Correlations  between  TSS  and  the 
two  aggregated  performance  measures  were  both  negative,  as  expected,  but  were  not  significant. 
Correlations  between  participants’  SSQ  and  their  SOT,  CCT,  and  PAC  scores  were  also 
evaluated.  Only  the  correlations  between  SSQ  and  attentional  control  were  consistent  and 
mostly  significant,  r  =  -0.612,  r  =  -0.425,  r  =  -0.432  (p  =  0.003,  p  =  0.035,  p  =  0.032)  for  nausea, 
oculomotor,  and  TSS,  respectively.  SSQ  for  higher  versus  lower  attentional  control  participants 
are  presented  in  table  3  and  figure  10. 
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Table  3.  Simulator  sickness  scores  (standard  deviations  are  shown  in  parentheses). 


Participants 

Nausea 

Oculomotor 

Disorientation 

TSS 

Low  Attentional 

29.68 

38.74 

24.75 

37.4 

Control 

(24.99) 

(31.13) 

(33.94) 

(31.95) 

High  Attentional 

14.31 

28.8 

15.31 

23.94 

Control 

(11.24) 

(13.75) 

(21.21) 

(13.45) 

Simulator  Sickness 

Effects  of  Perceived  Attentional  Control 


Nausea  Disorientation  Oculomotor  TSS 


SSQ  Sub  Scales  and  TSS 


Attention  al  Control 

■ - « -  Low 

II  High 


Error  Bars  show  95.0%  Cl  of  Mean 
Bars  show  Means 


Figure  10.  Simulator  sickness  and  effects  of  PAC. 


3.5  Usability  Questionnaire 

Participants’  ratings  and  their  comments  are  presented  in  appendix  G.  Generally,  the  TCU  was 
perceived  to  be  user  friendly.  However,  a  number  of  participants  suggested  that  it  required  too 
many  steps  to  complete  some  simple  tasks  and  they  needed  to  go  to  different  screens  to  complete 
those  steps  (e.g.,  putting  targets  on  the  map,  labeling  the  targets,  and  sending  spot  reports). 

While  they  were  completing  those  steps,  they  could  not  effectively  monitor  the  gunnery  station. 
If  these  steps  can  be  more  consolidated  and  centralized,  this  part  of  the  robotic  task  would  not 
require  as  much  visual  attention  as  it  currently  does. 
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4.  Discussion 


In  this  study,  we  simulated  an  MCS  environment  and  perfonned  an  experiment  to  examine  the 
workload  and  performance  of  the  combined  position  of  gunner  and  robotic  operator.  Results 
showed  that  gunner’s  target  detection  perfonnance  degraded  significantly  when  s/he  had  to 
concurrently  monitor,  manage,  or  teleoperate  a  UGV.  The  gunner’s  perfonnance  in  the  three 
concurrent  task  conditions  was  significantly  different  from  one  another,  with  the  Monitor 
condition  being  the  highest  and  Teleop  condition  being  the  lowest.  Participants’  SpA  as 
measured  by  their  SOT  scores  was  found  to  be  an  accurate  predictor  of  their  gunnery 
performance.  These  results  suggest  that,  if  it  is  necessary  for  the  gunner  to  concurrently  access 
information  from  the  robotic  assets,  the  robotic  tasks  should  be  limited  to  activities  such  as 
monitoring.  If  excessive  manipulation  of  the  user  interface  is  required  as  in  the  UGV  and  Teleop 
conditions,  gunnery  performance  will  be  significantly  affected.  When  selecting  personnel  for 
these  tasks,  it  might  be  beneficial  to  take  into  account  their  SpA.  Thomas  and  Wickens  (2004) 
showed  that  there  were  individual  differences  in  scanning  effectiveness  and  its  associated  target 
detection  performance.  However,  Thomas  and  Wickens  (2004)  did  not  examine  the 
characteristics  of  those  participants  who  had  more  effective  scanning  strategies.  Results  of  the 
current  study  suggest  that  spatial  tests  such  as  the  SOT  might  be  useful  in  examining  individual 
differences  in  scanning  behavior  and  target  detection  perfonnance.  Further  research  should  also 
examine  the  difference  between  the  SOT  and  the  CCT.  Based  on  the  results  of  the  cunent  study, 
it  appears  that  these  two  tests  tapped  different  aspects  of  SpA.  Some  cognitive  modeling  work 
done  by  ARL  researchers  suggested  that  the  CCT  may  reflect  ability  more  associated  with 
feature  comparison  than  with  spatial  rotation  (Kelley,  Wiley,  &  Lee,  2000).  Verbal  protocol 
obtained  from  research  participants  in  Kelley,  Wiley,  and  Lee  (2000)  indicated  that  participants 
used  a  variety  of  problem-solving  strategies  for  their  spatial  rotation  test  instead  of  mentally 
rotating  the  images. 

For  the  robotic  tasks,  there  were  significant  differences  among  the  Monitor,  UGV,  and  Teleop 
conditions  in  human  target  detection  performance,  with  the  UGV  being  the  lowest  (only  53%  were 
detected).  The  inferior  performance  associated  with  the  semi-autonomous  UGV  seemed  to  reflect 
participant’s  over-reliance  on  the  ATR  capabilities  and  failure  to  detect  more  targets  along  the 
route  that  were  not  detected  by  the  ATR.  In  contrast,  in  Chen  et  al.  (2005),  participants  had  the 
lowest  target  detection  using  the  Teleop.  However,  in  Chen  et  al.’s  (2005)  UGV  condition,  the 
ATR  capabilities  were  not  available.  Results  of  the  current  study  are  consistent  with  automation 
research  that  operators  may  develop  over-reliance  on  the  automatic  system  and  this  complacency 
may  negatively  affect  their  task  performance  (Parasuraman,  Molloy,  &  Singh,  1993).  Thomas  and 
Wickens  (2000)  showed  that  when  participants  had  access  to  information  gathered  from 
automatically  panning  cameras,  they  tended  to  prematurely  close  the  automatic  panning  feature 
before  finishing  examining  the  entire  environment.  On  the  other  hand,  participants  manually 
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panning  the  cameras  had  significantly  higher  target  detection  performance,  which  indicated  more 
adequate  panning.  It  is  worth  noting  that  these  findings  along  with  the  results  of  the  current  study 
do  not  necessarily  suggest  that  manual  manipulation  of  sensor  devices  be  used  instead  of  ATR  or 
auto-panning  devices.  However,  the  issue  of  over-reliance  on  these  automatic  capabilities  needs  to 
be  taken  into  account  when  one  is  designing  the  user  interface  where  these  features  are  one  of  the 
components. 

When  participants  only  had  to  operate  the  UGV  (i.e.,  UGV  Baseline),  their  overall  target  detection 
rate  was  80%;  when  they  had  to  concurrently  operate  the  UGV  and  perform  the  gunner’s  tasks 
(i.e.,  UGV  condition),  their  target  detection  using  the  UGV  dropped  to  67%.  It  also  appears  that 
the  perfonnance  difference  between  UGV  and  Teleop  widened  in  the  concurrent  task  conditions 
compared  with  the  single -task  (baseline)  conditions.  In  other  words,  the  UGV  Baseline  and 
Teleop  Baseline  were  at  similar  levels  but  UGV-Concurrent  was  significantly  lower  than  Teleop- 
Concurrent.  These  results  suggest  that  as  operator’s  tasks  become  more  challenging  (i.e., 
concurrent  conditions),  s/he  may  rely  more  on  the  ATR  capabilities  to  relieve  the  workload  if  they 
are  available.  However,  the  over-reliance  on  the  ATR  capabilities  may  result  in  overall 
performance  degradation  as  discussed  in  the  previous  paragraph. 

None  of  the  individual  difference  factors  was  found  to  significantly  correlate  with  participants’ 
robotic  task  performance.  The  lack  of  correlation  between  SpA  and  robotic  task  performance 
was  unexpected.  Chen  et  al.  (2005)  showed  that  those  with  higher  SpA  (as  measured  by  the 
CCT)  had  significantly  higher  performance  in  their  target  detection  task  using  the  UGV.  Further 
research  is  needed  to  examine  the  relationship  between  SpA  and  robotic  task  perfonnance. 

Participants’  communication  task  performance  degraded  when  their  robotic  tasks  became  more 
challenging  (i.e.,  UGV  and  Teleop  conditions).  It  is  interesting  to  note  that  participants  appeared 
to  be  able  to  perfonn  their  communication  task  at  similar  levels  in  the  Gunner  Baseline  and 
Monitor  conditions.  This  suggests  that  in  the  Monitor  condition,  participants  had  sufficient 
cognitive  resources  left  to  perfonn  the  communication  tasks.  Participants  with  higher  PAG 
(measured  by  a  self-assessment  survey)  performed  better  on  a  concurrent  communication  task, 
although  they  performed  at  a  similar  level  on  their  gunnery  and  robotic  control  tasks  as  those 
with  lower  PAG.  These  results  suggest  that  participants  devoted  most  of  their  attention  resources 
to  the  gunnery  and  robotic  tasks,  and  only  those  with  higher  attention  allocation  skills  could 
more  successfully  perform  the  tertiary  communication  tasks.  Since  communication  will  be  a 
critical  part  in  the  task  environment,  these  results  may  have  important  implications  for  personnel 
selection  for  the  Army’s  future  forces. 

Participants’  perceived  workload  increased  almost  linearly  in  order  from  the  Gunner  Baseline, 
Monitor,  UGV  and  to  the  Teleop  condition,  and  the  differences  among  the  four  conditions  were 
all  statistically  significant.  These  results  are  consistent  with  Schipani  (2003),  which  evaluated 
robotic  operator  workload  in  a  field  setting.  Although  many  of  the  ground  robotic  assets  in  the 
Army’s  FCS  program  will  be  semi-autonomous,  it  is  very  likely  that  teleoperation  will  be 
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required  at  times  when  the  robotic  assets  encounter  problems.  The  higher  workload  associated 
with  teleoperation  needs  to  be  taken  into  account  when  one  is  designing  the  user  interfaces  for 
the  FCS.  Additionally,  it  appears  that  participants  with  higher  SpA,  although  performing  better 
on  the  tasks,  did  not  perceive  the  tasks  as  less  demanding.  In  fact,  the  correlations  between 
participants’  perceived  workload  and  their  CCT  scores  or  SOT  reaction  times  indicated  that 
those  with  higher  SpA  (at  least  according  to  these  two  measures)  actually  perceived  the  tasks  as 
more  demanding.  The  correlations  between  participants’  PAC  and  their  workload  were  as 
expected.  Those  with  higher  PAC  thought  the  tasks  were  less  demanding.  It  is  worth  noting  that 
only  in  the  most  challenging  condition  (i.e..  Teleop)  did  the  correlation  reach  significance.  In 
other  words,  as  the  tasks  became  more  difficult,  the  differences  in  the  levels  of  perceived 
workload  between  those  with  higher  and  lower  PAC  appeared  to  widen.  The  relationships 
between  workload  and  PAC  and  SpA  need  to  be  further  investigated. 

Participants  seemed  to  experience  fairly  significant  levels  of  simulator  sickness,  especially  in  the 
oculomotor  area.  The  high  demand  of  visual  attention,  particularly  in  the  concurrent  task 
conditions,  may  have  contributed  to  the  elevated  levels  of  simulator  sickness  the  participants 
experienced.  It  is  worth  noting  that  our  entire  experimental  session  only  lasted  about  1-1/2  hours. 
Any  duration  longer  than  this  may  induce  even  more  severe  sickness.  Participants  with  lower  PAC 
had  significantly  higher  simulator  sickness  than  did  those  with  higher  attentional  control.  It  was 
reported  that  in  the  virtual  environment,  those  who  need  to  concentrate  more  tended  to  experience 
higher  levels  of  simulator  sickness  (Regan,  as  cited  in  Kolasinski,  1995).  The  findings  of  the 
Regan  study  and  the  current  study  seem  to  suggest  that  those  who  are  better  at  allocating  their 
attentional  resources  may  experience  lower  simulator  sickness.  Alternatively,  those  with  lower 
PAC  might  pay  more  attention  to  (and  be  distracted  by)  their  own  bodily  reaction  (Jerome, 
personal  communication,  February  10,  2006).  This  increased  awareness  may  have  contributed  to 
the  elevated  levels  of  simulator  sickness.  It  is  also  possible  that  they  were  not  experiencing  more 
sickness  but  were  simply  more  aware  of  the  symptoms.  More  research  in  this  area  is  needed  to 
examine  this  relationship. 


5.  Conclusions 


The  findings  of  the  current  study  suggest  that  in  an  FCS  task  environment  such  as  the  MCS,  if  it  is 
necessary  for  the  gunner  to  concurrently  access  information  from  the  robotic  assets,  the  robotic 
tasks  should  be  limited  to  activities  such  as  monitoring.  If  excessive  manipulation  of  the  user 
interface  occurs  as  in  the  UGV  and  Teleop  conditions,  gunnery  performance  and  communications 
with  fellow  crew  members  may  be  significantly  affected.  When  selecting  personnel  for  these 
tasks,  it  might  be  beneficial  to  take  into  account  their  SpA  (SOT)  and  PAC.  In  the  current  study, 
those  with  higher  SpA  as  measured  by  the  SOT  appeared  to  be  better  able  to  perfonn  their  gunnery 
tasks  concurrently  with  the  other  two  tasks.  Those  with  higher  PAC  were  more  effective  in 
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performing  the  communicating  tasks  concurrently  with  the  tasks,  especially  when  the  robotic  tasks 
needed  more  attention  and  manipulations.  They  also  experienced  a  lower  level  of  simulator 
sickness. 
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Appendix  A.  Demographic  Questionnaire 


Participant# _  Age _  Major _  Date _  Gender _ 

1.  What  is  the  highest  level  of  education  you  have  had? 

Less  than  4  yrs  of  college _  Completed  4  yrs  of  college _  Other _ 

2.  When  did  you  use  computers  in  your  education?  ( Circle  all  that  apply ) 

Grade  School  Jr.  High  High  School 

Technical  School  College  Did  Not  Use 

3.  Where  do  you  currently  use  a  computer?  (Circle  all  that  apply 1 

Home  Work  Library  Other _  Do  Not  Use 

4.  For  each  of  the  following  questions,  circle  the  response  that  best  describes  you. 

How  often  do  you: 

Use  a  mouse?  Daily,  Weekly,  Monthly,  Once  every  few  months,  Rarely,  Never 

Use  a  joystick?  Daily,  Weekly,  Monthly,  Once  every  few  months.  Rarely,  Never 

Use  a  touch  screen?  Daily,  Weekly,  Monthly,  Once  every  few  months,  Rarely,  Never 

Use  icon-based  programs/software? 

Daily,  Weekly,  Monthly,  Once  every  few  months.  Rarely,  Never 
Use  programs/software  with  pull-down  menus? 

Daily,  Weekly,  Monthly,  Once  every  few  months.  Rarely,  Never 
Use  graphics/drawing  features  in  software  packages? 

Daily,  Weekly,  Monthly,  Once  every  few  months.  Rarely,  Never 
Use  E-mail?  Daily,  Weekly,  Monthly,  Once  every  few  months,  Rarely,  Never 

Operate  a  radio  controlled  vehicle  (car,  boat,  or  plane)? 

Daily,  Weekly,  Monthly,  Once  every  few  months.  Rarely,  Never 

Play  computer/video  games? 

Daily,  Weekly,  Monthly,  Once  every  few  months.  Rarely,  Never 

5.  Which  type(s)  of  computer/video  games  do  you  most  often  play  if  you  play  at  least  once  every  few  months? 

6.  Which  of  the  following  best  describes  your  expertise  with  computer?  (check  V  one) 

_ Novice 

_ Good  with  one  type  of  software  package  (such  as  word  processing  or  slides) 

_ Good  with  several  software  packages 

_ Can  program  in  one  language  and  use  several  software  packages 

_ Can  program  in  several  languages  and  use  several  software  packages 

7.  Are  you  in  your  usual  state  of  health  physically?  YES  NO 
If  NO,  please  briefly  explain: 

8.  How  many  hours  of  sleep  did  you  get  last  night? _ hours 

9.  Do  you  have  normal  color  vision?  YES  NO 

10.  Do  you  have  prior  military  service?  YES  NO  If  Yes,  how  long _ 
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Appendix  B.  Attentional  Control  Survey 


For  each  of  the  following  questions,  circle  the  response  that  best  describes  you. 

It  is  very  hard  for  me  to  concentrate  on  a  difficult  task  when  there  are  noises  around. 

Almost  never,  Sometimes,  Often,  Always 

When  I  need  to  concentrate  and  solve  a  problem,  I  have  trouble  focusing  my  attention. 

Almost  never,  Sometimes,  Often,  Always 

When  I  am  working  hard  on  something,  I  still  get  distracted  by  events  around  me. 

Almost  never,  Sometimes,  Often,  Always 

My  concentration  is  good  even  if  there  is  music  in  the  room  around  me. 

Almost  never,  Sometimes,  Often,  Always 

When  concentrating,  I  can  focus  my  attention  so  that  1  become  unaware  of  what’s  going  on  in  the  room  around 
me.  Almost  never,  Sometimes,  Often,  Always 

When  I  am  reading  or  studying,  I  am  easily  distracted  if  there  are  people  talking  in  the  same  room. 

Almost  never,  Sometimes,  Often,  Always 

When  trying  to  focus  my  attention  on  something,  I  have  difficulty  blocking  out  distracting  thoughts. 

Almost  never,  Sometimes,  Often,  Always 

I  have  a  hard  time  concentrating  when  I’m  excited  about  something. 

Almost  never,  Sometimes,  Often,  Always 

When  concentrating,  I  ignore  feelings  of  hunger  or  thirst.  Almost  never,  Sometimes,  Often,  Always 

I  can  quickly  switch  from  one  task  to  another.  Almost  never,  Sometimes,  Often,  Always 

It  takes  me  a  while  to  get  really  involved  in  a  new  task.  Almost  never,  Sometimes,  Often,  Always 

It  is  difficult  for  me  to  coordinate  my  attention  between  the  listening  and  writing  required  when  taking  notes 
during  lectures.  Almost  never,  Sometimes,  Often,  Always 

I  can  become  interested  in  a  new  topic  very  quickly  when  1  need  to. 

Almost  never,  Sometimes,  Often,  Always 

It  is  easy  for  me  to  read  or  write  while  I’m  also  talking  on  the  phone. 

Almost  never,  Sometimes,  Often,  Always 

I  have  trouble  carrying  on  two  conversations  at  once.  Almost  never,  Sometimes,  Often,  Always 

I  have  a  hard  time  coming  up  with  new  ideas  quickly.  Almost  never,  Sometimes,  Often,  Always 

After  being  interrupted  or  distracted,  I  can  easily  shift  my  attention  back  to  what  I  was  doing  before. 

Almost  never,  Sometimes,  Often,  Always 

When  a  distracting  thought  comes  to  mind,  it  is  easy  for  me  to  shift  my  attention  away  from  it. 

Almost  never,  Sometimes,  Often,  Always 

It  is  easy  for  me  to  alternate  between  two  different  tasks.  Almost  never,  Sometimes,  Often,  Always 

It  is  hard  for  me  to  break  from  one  way  of  thinking  about  something  and  look  at  it  from  another  point  of  view. 

Almost  never,  Sometimes,  Often,  Always 
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Appendix  C.  NASA-TLX  Questionnaire 


Please  rate  your  overall  impression  of  demands  imposed  on  you  during  the  exercise. 

1.  Mental  Demand:  How  much  mental  and  perceptual  activity  was  required  (e.g.,  thinking,  looking,  searching, 
etc.)?  Was  the  task  easy  or  demanding,  simple  or  complex,  exacting  or  forgiving? 

LOW  l-l-l-l-H-l-l-H  HIGH 
123456789  10 

2.  Physical  Demand:  How  much  physical  activity  was  required  (e.g.,  pushing,  pulling,  turning,  controlling, 
activating,  etc.)?  Was  the  task  easy  or  demanding,  slow  or  brisk,  slack  or  strenuous,  restful  or  laborious? 

LOW  l-l-l-l-H-l-l-H  high 
123456789  10 

3.  Temporal  Demand:  How  much  time  pressure  did  you  feel  due  to  the  rate  or  pace  at  which  the  task  or  task 
elements  occurred?  Was  the  pace  slow  and  leisurely  or  rapid  and  frantic? 

LOW  l-l-l-l-l-l-l-l-l-l  HIGH 
123456789  10 

4.  Level  of  Effort:  How  hard  did  you  have  to  work  (mentally  and  physically)  to  accomplish  your  level  of 
performance? 


LOW  l-l-l-l-l-l-l-l-l-l  HIGH 
123456789  10 

5.  Level  of  Frustration:  How  insecure,  discouraged,  irritated,  stressed  and  annoyed  versus  secure,  gratified,  content, 
relaxed  and  complacent  did  you  feel  during  the  task? 

LOW  l-l-l-l-l-l-l-l-l-l  HIGH 
123456789  10 

6.  Performance:  How  successful  do  you  think  you  were  in  accomplishing  the  goals  of  the  task  set  by  the 
experimenter  (or  yourself)?  How  satisfied  were  you  with  your  performance  in  accomplishing  these  goals? 

LOW  l-l-l-l-l-l-l-l-l-l  HIGH 
123456789  10 

7.  Please  mark  the  indicated  loading  that  most  closely  matches  the  work  performed  by  your  visual,  cognitive,  and 
motor  efforts  on  the  task  just  completed. 

Visual  LOW  Load  |— |— |— |— |— |— |— |— |— |  HIGH  Load 

123456789  10 

Cognitive  LOW  Load  |— |— |— |— |— |— |— |— |— |  HIGH  Load 

123456789  10 

Psvchomotor  (Relating  to  the  physical  activities  associated  with  mental  processes 

LOW  Load  |— |— |— |— |— |— |— |— |— |  HIGH  Load 
123456789  10 
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Appendix  D.  Simulator  Sickness  (Current  Health  Status)  Questionnaire 


ID _  Time  &  Date 

Instructions:  Please  indicate  how  you  feel  right  now  in  the  following  areas,  by  circling  the 
word  that  applies. 


1. 

General  Discomfort 

None 

Slight 

Moderate 

Severe 

2. 

Fatigue 

None 

Slight 

Moderate 

Severe 

3. 

Headache 

None 

Slight 

Moderate 

Severe 

4. 

Eye  Strain 

None 

Slight 

Moderate 

Severe 

5. 

Difficulty  Focusing 

None 

Slight 

Moderate 

Severe 

6. 

Increased  Salivation 

None 

Slight 

Moderate 

Severe 

7. 

Sweating 

None 

Slight 

Moderate 

Severe 

8. 

Nausea 

None 

Slight 

Moderate 

Severe 

9. 

Difficulty  Concentrating 

None 

Slight 

Moderate 

Severe 

10. 

Fullness  of  Head 

None 

Slight 

Moderate 

Severe 

11. 

Blurred  vision 

None 

Slight 

Moderate 

Severe 

12. 

Dizzy  (Eyes  Open) 

None 

Slight 

Moderate 

Severe 

13. 

Dizzy  (Eyes  Closed) 

None 

Slight 

Moderate 

Severe 

14. 

Vertigo* 

None 

Slight 

Moderate 

Severe 

15. 

Stomach  Awareness** 

None 

Slight 

Moderate 

Severe 

16. 

Burping 

None 

Slight 

Moderate 

Severe 

Vertigo  is  a  disordered  state  in  which  the  person  or  his/her  surroundings  seem  to  whirl  dizzily:  giddiness. 
Stomach  awareness  is  usually  used  to  indicate  a  feeling  of  discomfort  which  is  just  short  of  nausea. 

Are  there  any  other  symptoms  you  are  experiencing  right  now?  If  so,  please  describe  the  symptom(s)  and 
rate  its/their  severity  below.  Use  the  other  side  if  necessary. 
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Appendix  E.  Usability  Questionnaire 


Instructions 


Usability  Survey 


Participant  # 


This  questionnaire  is  designed  to  elicit  your  reactions  for  the  use  of  the  OCU.  Your  responses  will  help  us 
understand  what  aspects  of  the  OCU  you  find  inefficient  or  difficult  to  use.  This  questionnaire  provides  an 
opportunity  to  evaluate  the  usability  of  the  system.  We  will  use  your  responses  to  influence  the  design  of  the  future 
OCU.  To  as  great  a  degree  as  possible,  think  about  all  the  tasks  that  you  have  performed  with  the  OCU  while  you 
answer  the  questions. 

Read  each  statement  or  question  and  indicate  your  response  to  the  statement  or  question  by  circling  the  value  you 
feel  best  gauges  your  answer  to  the  question  on  the  scale  underneath  it.  If  a  statement  does  not  apply,  circle  N/A. 
Use  the  space  on  the  right  to  provide  any  comments  that  would  help  us  improve  the  design  of  the  crew  station 
interface.  Comments  clarifying  answers  are  extremely  useful. 


Asset  Summary 


Comments 

1.  There  was  adequate  information  displayed  with  regard  to  the  state  of  the  Unmanned 
Vehicles  (UVs). 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 
1  2  3  4  5  6  7 


2.  The  text  on  the  Asset  Summary  screen  was  easy  to  read. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 
1  2  3  4  5  6  7 


3.  The  information  presented  in  the  display  was  easy  to  interpret. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 


4.  Gaining  control  of  an  UV  was  easy  to  do. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 


5.  The  Asset  Summary  display  was  easy  to  use. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 


Map  Display 

1.  The  button  labels  accurately  represented  the  buttons'  functionality. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 
1  2  3  4  5  6  7 


2.  The  information  presented  on  the  mapping  display  was  easy  to  read. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 


Comments 


33 


Comments 


3.  The  map  resolution  made  it  difficult  for  me  to  select  a  specific  location. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 


4.  It  was  easy  to  understand  what  the  icons  used  on  the  map  displays  represented. 
Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 


5.  I  was  able  to  navigate  the  map  effectively. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 
1  2  3  4  5  6  7 


6.  Icons  representing  different  unmanned  vehicles  were  easy  to  distinguish. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 


7.  The  map  terrain  features  (roads,  water,  vegetation)  were  difficult  to  see. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 
1  2  3  4  5  6  7 


8.  The  orientation  of  the  unmanned  vehicles  is  represented  clearly. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 


9.  The  map  was  my  main  source  of  situation  awareness. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 


10.  Text  on  the  map  status  display  was  easy  to  understand. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 
1  2  3  4  5  6  7 


11.  It  was  easy  to  modify  the  icons  on  the  map. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 
1  2  3  4  5  6  7 


12.  Overall  the  mapping  display  was  easy  to  use. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 
1  2  3  4  5  6  7 


RSTA 

1.  The  print/text  used  on  the  RSTA  display  was  easy  to  read. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 
1  2  3  4  5  6  7 


2.  The  layout  of  the  RSTA  screen  made  sense. 

Strongly  DISAGREE  | — |— -| — | — | — | — |  Strongly  AGREE  N/A 
1  2  3  4  5  6  7 


3.  The  video  imagery  for  RSTA  Scan  from  the  XUVs  is  clear. 

Strongly  DISAGREE  | — | — |— -| — | — | — |  Strongly  AGREE  N/A 
1  2  3  4  5  6  7 
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Comments 


4.  The  RSTA  display  should  contain  additional  information. 

Strongly  DISAGREE  | — |— -|— -| — | — | — |  Strongly  AGREE  N/A 
1  2  3  4  5  6  7 


5.  The  button  labels  accurately  represented  their  intended  functions. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 


6.  I  would  like  to  see  more  functionality  added  to  the  RSTA  Scan. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 
1  2  3  4  5  6  7 


7.  This  display  made  it  easy  to  queue  targets. 

Strongly  DISAGREE  | — |— -|— -| — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 


8.  I  was  able  to  manipulate  the  RSTA  images  effectively. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 


Teleoperation 

1.  The  streaming  video  was  sufficient  to  allow  me  to  teleoperate  the  XUV. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 
1  2  3  4  5  6  7 

2.  I  used  the  map  display  while  teleoperating  an  XUV  to  help  me  maintain  the  location 
of  an  XUV. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 
1  2  3  4  5  6  7 


3.  There  was  a  significant  time  delay  that  made  controlling  the  XUV  difficult. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 


4.  The  button  labels  accurately  represented  the  buttons'  functionality. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 

5.  I  found  the  Teleoperation  display  easy  to  use  to  control  XUVs. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 

Reporting 

1.  The  button  labels  accurately  represented  the  buttons'  functionality. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 


Comments 


Comments 
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2. 


Comments 


Much  of  the  information  presented  on  the  display  was  NOT  helpful. 

Strongly  DISAGREE  | — | — | — | — |— -| — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 


3.  I  sometimes  felt  'lost'  navigating  the  reporting  display  screens. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 
1  2  3  4  5  6  7 


4.  There  was  not  enough  information  displayed  on  the  reporting  screen. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 
1  2  3  4  5  6  7 


5.  Composing  a  SPOT  report  was  an  easy  task  to  accomplish. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 


6.  Composing  a  SPOT  report  took  too  long  to  accomplish. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 
1  2  3  4  5  6  7 

7.  The  entry  fields  for  composing  a  SPOT  report  were  found  in  the  order  I  expected  them 
to  be. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 
1  2  3  4  5  6  7 


8.  Using  the  reporting  display  to  compose  reports  was  an  easy  process  overall. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 
1  2  3  4  5  6  7 


General  Usability-  RCTA  OCU 

Definitions: 

•  A  screen  is  defined  as  the  physical  object  that  may  be  handled. 

•  A  display  is  the  software  image  that  is  displayed  on  the  screen. 

•  A  soft  button  is  a  computer  generated  image  of  a  button  displayed  on  the  screen 


1.  The  touchscreen  buttons  were  large  enough  to  use  effectively. 

Strongly  DISAGREE  | — |— -| — | — | — | — |  Strongly  AGREE  N/A 
1  2  3  4  5  6  7 


2.  The  print/  text  on  the  OCU  soft  buttons  was  clear,  and  easy  to  read. 

Strongly  DISAGREE  | — |—  -| — | — | — | — |  Strongly  AGREE  N/A 
1  2  3  4  5  6  7 
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Comments 


3.  It  was  easy  to  learn  to  use  the  basic  features  of  the  OCU. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 
1  2  3  4  5  6  7 


4.  There  are  not  enough  displays  to  accomplish  the  required  tasks. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 
1  2  3  4  5  6  7 

5.  The  point  at  which  my  finger  touches  is  not  identical  to  the  point  indicated  by  the 
touchscreen  (there  are  parallax  problems). 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 
1  2  3  4  5  6  7 

6.  Throughout  the  OCU,  the  same  terminology  is  used  to  indicate  the  same 
information. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 
1  2  3  4  5  6  7 


7.  I  sometimes  feel  "lost"  when  working  with  the  OCU. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 


8.  I  had  a  hard  time  finding  information  I  needed  on  the  OCU. 

Strongly  DISAGREE  | — |— -| — |— -| — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 


9.  It  was  easy  to  understand  what  the  icons  used  on  the  OCU  displays  represented. 
Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 


10. 


The  OCU  gives  appropriate  warning  messages  when  I  am  about  to  make  a  serious 
mistake. 


Strongly  DISAGREE  | — | — | — | — |— -| — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 


11.  With  the  OCU,  if  I  make  a  mistake  I  can  correct  it  easily. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 


12.  It  is  obvious  which  command  button  brings  up  the  OCU  display  I  need  next. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 


13.  Information  is  appropriately  arranged  on  the  OCU  displays. 

Strongly  DISAGREE  | — — | — | — |  Strongly  AGREE  N/A 
1  2  3  4  5  6  7 


14.  Color-coding  on  the  OCU  displays  is  helpful. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 


15.  The  symbols  that  were  used  were  hard  to  learn. 

Strongly  DISAGREE  |— -| — |— -| — |— -| — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 
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16.  Overall,  I  think  the  OCU  was  easy  to  learn. 

Strongly  DISAGREE  | — |—  -| — | — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 


17.  Overall,  I  think  the  OCU  was  easy  to  work  with. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 

1  2  3  4  5  6  7 
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Appendix  F.  Scoring  Procedure  for  the  SSQ 


Symptoms  scored  0  (None)  -  3  (Severe) 

Nausea  (Raw)  -  Sum  of  General  discomfort,  increased  salivation,  sweating,  nausea,  diff 
concentrating,  stomach  awareness,  burping 

Nausea  sub  scale  =  Nausea  (Raw)  x  9.54 

Oculomotor  -  Sum  of  general  discomfort,  fatigue,  headache,  eye  strain,  diff  focusing,  diff 
concentrating,  blurred  vision 

Oculomotor  sub  scale  =  Nausea  (Raw)  x  7.58 

Disorientation  -  Sum  of  diff  focusing,  nausea,  fullness  of  head,  blurred  vision,  dizzy 
(eyes  open),  dizzy  (eyes  closed),  vertigo 

Disorientation  sub  scale  =  Nausea  (Raw)  x  13.92 

TSS  =  [Nausea  (Raw)  +  Oculomotor  (Raw)  +  Disorientation  (Raw)]  x  3.74 
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Appendix  G.  Results  of  the  Usability  Questionnaire  and  Selected  Comments 
from  Participants 


Note: 

1.  Scale: 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  N/A 
1  2  3  4  5  6  7 

2.  The  average  of  participants  ratings  are  presented  in  the  right  column. 

3.  Participants’  Comments  are  presented  after  each  question  in  orange  ink. 

4.  Additional  general  comments  are  at  the  bottom  of  this  appendix. 

Asset  Summary 

Rating 

6.  There  was  adequate  information  displayed  with  regard  to  the  state  of  the  5.4 

Unmanned  Vehicles  (UVs). 

Much  of  which  had  no  impact  on  actions 

7.  The  text  on  the  Asset  Summary  screen  was  easy  to  read.  5.97 

Might  be  better  to  group  snapshots  and  mosaic  together  side-by-side 

Grab  snapshot  ->  Exec  Snapshot 
Grab  Mosaic  ->  Exec  Mosaic 

Probably  just  takes  more  getting  used  to,  unclear  at  first 

8.  The  information  presented  in  the  display  was  easy  to  interpret.  5.67 

But  there  was  quite  a  lot  to  interpret 

9.  Gaining  control  of  an  UV  was  easy  to  do.  4.72 

10.  The  Asset  Summary  display  was  easy  to  use.  5.47 

Probably  just  takes  more  getting  used  to,  unclear  at  first 

Map  Display 

13.  The  button  labels  accurately  represented  the  buttons’  functionality.  Rating 

A  lot  of  unused  buttons  5.94 

14.  The  information  presented  on  the  mapping  display  was  easy  to  read.  5.72 

15.  The  map  resolution  made  it  difficult  for  me  to  select  a  specific  location.  3.89 

Good  res 

16.  It  was  easy  to  understand  what  the  icons  used  on  the  map  displays  5.61 

represented. 

17.  I  was  able  to  navigate  the  map  effectively.  5.67 

Perhaps  consider  changing  color  of  route  to  different  than  rivers 

18.  Icons  representing  different  unmanned  vehicles  were  easy  to  distinguish.  5.36 

19.  The  map  terrain  features  (roads,  water,  vegetation)  were  difficult  to  see.  3.64 

20.  The  orientation  of  the  unmanned  vehicles  is  represented  clearly.  5.5 
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21.  The  map  was  my  main  source  of  situation  awareness.  5.44 

22.  Text  on  the  map  status  display  was  easy  to  understand.  5.5 

23.  It  was  easy  to  modify  the  icons  on  the  map.  4.35 

Too  many  different  screens  to  accomplish  one  task 

24.  Overall  the  mapping  display  was  easy  to  use.  5.75 

RSTA 

9.  The  print/text  used  on  the  RSTA  display  was  easy  to  read.  Rating 

5.83 

10.  The  layout  of  the  RSTA  screen  made  sense.  5.64 

After  a  little  use,  easy  to  comprehend 

Biggest  difficulty  JPEG  artifact  impeded  target  ID 

11.  The  video  imagery  for  RSTA  Scan  from  the  XUVs  is  clear.  3.53 

Res  wasn’t  very  good,  had  to  take  multiple  mosaics/pics  to  be  sure 

12.  The  RSTA  display  should  contain  additional  information.  4.33 

Which  checkpoint  you  are  at 

#  of  enemies  queued  so  far 

Perhaps  could  add  distance  or  thermal  imaging 

13.  The  button  labels  accurately  represented  their  intended  functions.  5.3 

Some  are  similar  in  their  labels  but  different  in  use  (edit  ->  rename,  lable) 

14.  I  would  like  to  see  more  functionality  added  to  the  RSTA  Scan. 

Useful  to  have  a  list  of  labeled  enemies,  that  way  you  can  number  them  in  order  4.83 

when  you  approach  another  enemy 
Pretty  good  amount 

Useful  to  know  if  a  single  target  was  queued  more  than  once  (or  “already”) 

15.  This  display  made  it  easy  to  queue  targets.  5.06 

Too  many  screens  to  accomplish  a  similar  and  simple  action 

16.  I  was  able  to  manipulate  the  RSTA  images  effectively.  5.08 

Teleoperation 

6.  The  streaming  video  was  sufficient  to  allow  me  to  teleoperate  the  XUV.  Rating 

Yes,  but...  had  to  get  very  close  to  inspect  images  to  ID  4.97 

7.  I  used  the  map  display  while  teleoperating  an  XUV  to  help  me  maintain  the  6.33 
location  of  an  XUV. 

Mainly  just  starting  and  stopping 

8.  There  was  a  significant  time  delay  that  made  controlling  the  XUV  difficult.  4.35 
Turning  seems  a  little  slow 

9.  The  button  labels  accurately  represented  the  buttons’  functionality.  5.67 

10.  I  found  the  Teleoperation  display  easy  to  use  to  control  XUVs.  4.72 

Reporting 

9.  The  button  labels  accurately  represented  the  buttons’  functionality.  Rating 

Though  the  button  path  to  report  seems  long  5.83 

Much  had  no  bearing  on  tasks  of  testing 
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10.  Much  of  the  information  presented  on  the  display  was  NOT  helpful.  4.44 

At  least  to  my  needs  of  the  experiment 

11.  I  sometimes  felt  ‘lost’  navigating  the  reporting  display  screens.  3.39 

Color  coding  the  buttons  may  help 

Especially  at  first,  recommend  simplifying  submitting  a  report  or  make  it  the  same 
for  both  RSTA  &  Teleop 

12.  There  was  not  enough  information  displayed  on  the  reporting  screen.  2.42 

13.  Composing  a  SPOT  report  was  an  easy  task  to  accomplish.  5.94 

After  practice,  yes,  but  should  try  to  keep  it  the  same  for  both  RSTA  &  Teleop 

Very  quick...  block  on  repeat  usage  of  status  was  odd... 

14.  Composing  a  SPOT  report  took  too  long  to  accomplish.  3.5 

At  least  when  operating  gunner  also 

15.  The  entry  fields  for  composing  a  SPOT  report  were  found  in  the  order  I  4.93 

expected  them  to  be. 

Should  try  to  keep  them  together,  or  in  order  of  use 
Vertical  flat  panel  wants  two  hands  for  rapid  typing 

16.  Using  the  reporting  display  to  compose  reports  was  an  easy  process  overall.  4.94 
After  practice,  yes 

General  Usability-  RCTA  OCU 

18.  The  touchscreen  buttons  were  large  enough  to  use  effectively.  Rating 

6.06 

19.  The  print/  text  on  the  OCU  soft  buttons  was  clear,  and  easy  to  read.  6.1 1 

20.  It  was  easy  to  learn  to  use  the  basic  features  of  the  OCU.  5.5 

21.  There  are  not  enough  displays  to  accomplish  the  required  tasks.  2.81 

22.  The  point  at  which  my  finger  touches  is  not  identical  to  the  point  2.75 

indicated  by  the  touchscreen  (there  are  parallax  problems). 

23.  Throughout  the  OCU,  the  same  terminology  is  used  to  indicate  the  4.89 


same  information. 

Double  “click”  of  chip  select  sometimes  irritaing 

3.44 

24.  I  sometimes  feel  “lost”  when  working  with  the  OCU. 

At  first 

Associate  double  click  of  “Get  Mosaic”  or  “Get  Snapshot”  seemed  to  break  things 


25.  I  had  a  hard  time  finding  information  I  needed  on  the  OCU.  Rating 

There  are  a  lot  of  displays  3.14 

26.  It  was  easy  to  understand  what  the  icons  used  on  the  OCU  displays  4.94 

represented. 

27.  The  OCU  gives  appropriate  warning  messages  when  I  am  about  to  2.83 

make  a  serious  mistake. 

Never  encountered  any 

Hit  wrong  button  clears  easily  by  hitting  right  button 
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28.  With  the  OCU,  if  I  make  a  mistake  I  can  correct  it  easily.  4.38 

29.  It  is  obvious  which  command  button  brings  up  the  OCU  display  I  need  4.72 

next. 

Except  when  generating  a  report,  then  you  need  multiple  displays 

30.  Information  is  appropriately  arranged  on  the  OCU  displays.  4.97 

31.  Color-coding  on  the  OCU  displays  is  helpful.  5.53 

“Selected”  color  change-  informative  but  not  emphatic 

32.  The  symbols  that  were  used  were  hard  to  learn.  2.72 

33.  Overall,  I  think  the  OCU  was  easy  to  learn.  5.11 

Took  awhile,  and  lots  of  errors 

34.  Overall,  I  think  the  OCU  was  easy  to  work  with.  5.28 


Graying  out  buttons  that  aren’t  useful  at  the  moment  can  be  helpful,  e.g.,  “execute 
plan”  button  after  the  XUV  is  started 
Yes,  after  the  steep  learning  curve  it  was 

General  Comments: 

•  Group  execute  button  near  snapshot  &  grab  mosaic 

•  Automate  labeling 

•  Gray  out  buttons  that  are  not  necessary  for  a  particular  function 

•  RSTA  scan  -  after  queue,  provide  pop  up  menu  to  label  and  report  on  RSTA  screen  instead  of 
switching  to  map 

•  Same  for  reports  -  should  not  have  to  switch  to  report  display 

•  Teleop  -  better  resolution 
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Intentionally  left  blank 
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Glossary  of  Acronyms 


ANOVA 

analysis  of  variance 

ARL 

Army  Research  Laboratory 

ATR 

aided  target  recognition 

BLOS 

beyond-line-of-sight 

CCT 

Cube  Comparison  Test 

FCS 

Future  Combat  System 

LOS 

line-of-sight 

LSD 

Least  Significant  Difference 

MCS 

Mounted  Combat  System 

OCU 

operator  control  unit 

OTW 

out-the-window 

PAC 

perceived  attentional  control 

RCTA 

Robotics  Collaborative  Technology  Alliance 

RSTA 

Reconnaissance,  Surveillance,  and  Target  Acquisition 

SA 

situational  awareness 

SIL 

System  Integration  Laboratory 

SNR 

signal-to-noise  ratio 

SOT 

Spatial  Orientation  Test 

SpA 

spatial  ability 

SSQ 

Simulator  Sickness  Questionnaire 

TCU 

Tactical  Control  Unit 

TS 

total  severity 

TSS 

Total  Severity  Score 

UCD 

unmanned  combat  demonstration 

UGV 

unmanned  ground  vehicle 

XUV 

experimental  unmanned  vehicle 
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1  DEFENSE  TECHNICAL 
(PDF  INFORMATION  CTR 
ONLY)  DTICOCA 

8725  JOHN  J  KINGMAN  RD 
STE  0944 

FORT  BELVOIR  VA  22060-6218 

1  US  ARMY  RSRCH  DEV  &  ENGRG  CMD 
SYSTEMS  OF  SYSTEMS 
INTEGRATION 
AMSRD  SS  T 
6000  6TH  ST  STE  100 
FORT  BELVOIR  VA  22060-5608 

1  INST  FOR  ADVNCD  TCHNLGY 
THE  UNIV  OF  TEXAS  AT  AUSTIN 
3925  W  BRAKER  LN  STE  400 
AUSTIN  TX  78759-5316 

1  DIRECTOR 

US  ARMY  RESEARCH  LAB 
IMNE  ALC  IMS 
2800  POWDER  MILL  RD 
ADELPHI  MD  20783-1197 

1  DIRECTOR 

US  ARMY  RESEARCH  LAB 
AMSRD  ARL  Cl  OK  TL 
2800  POWDER  MILL  RD 
ADELPHI  MD  20783-1197 

2  DIRECTOR 

US  ARMY  RESEARCH  LAB 
AMSRD  ARL  Cl  OK  T 
2800  POWDER  MILL  RD 
ADELPHI  MD  20783-1197 

1  ARMY  RSCH  LABORATORY  -  HRED 
ATTN  AMSRD  ARL  HR  M  DR  M  STRUB 
6359  WALKER  LANE  SUITE  100 
ALEXANDRIA  VA  22310 

1  ARMY  RSCH  LABORATORY  -  HRED 
ATTN  AMSRD  ARL  HR  MA  J  MARTIN 
MYER  CENTER  RM  2D3 1 1 
FT  MONMOUTH  NJ  07703-5630 

1  ARMY  RSCH  LABORATORY  -  HRED 
ATTN  AMSRD  ARL  HR  MC  A  DAVISON 
320  MANSCEN  LOOP  STE  166 
FT  LEONARD  WOOD  MO  65473-8929 


1  ARMY  RSCH  LABORATORY  -  HRED 
ATTN  AMSRD  ARL  HR  MD  T  COOK 
BLDG  5400  RM  C242 
REDSTONE  ARSENAL  AL  35898-7290 

1  COMMANDANT  USAADASCH 
ATTN  ATS  A  CD 

ATTN  AMSRD  ARL  HR  ME  MS  A  MARES 
5800  CARTER  RD 
FT  BLISS  TX  79916-3802 

1  ARMY  RSCH  LABORATORY  -  HRED 

ATTN  AMSRD  ARL  HR  MI  J  MINNINGER 

BLDG  5400  RM  C242 

REDSTONE  ARSENAL  AL  35898-7290 

1  ARMY  RSCH  LABORATORY  -  HRED 
ATTN  AMSRD  ARL  HR  MM  DR  V  RICE 
BLDG  4011  RM  217 
1750  GREELEY  RD 
FT  SAM  HOUSTON  TX  78234-5094 

1  ARMY  RSCH  LABORATORY  -  HRED 
ATTN  AMSRD  ARL  HR  MG  R  SPINE 
BUILDING  333 

PICATINNY  ARSENAL  NJ  07806-5000 

1  ARMY  RSCH  LABORATORY  -  HRED 
ATTN  AMSRD  ARL  HR  MH  C  BURNS 
BLDG  1002  ROOM  117 
1ST  CAVALRY  REGIMENT  RD 
FT  KNOX  KY  40121 

1  ARMY  RSCH  LABORATORY  -  HRED 
AVNC  FIELD  ELEMENT 
ATTN  AMSRD  ARL  HR  MJ  D  DURBIN 
BLDG  4506  (DCD)  RM  107 
FT  RUCKER  AL  36362-5000 

1  ARMY  RSCH  LABORATORY  -  HRED 

ATTN  AMSRD  ARL  HR  MK  MR  J  REINHART 
10125  KINGMAN  RD 
FT  BELVOIR  VA  22060-5828 

1  ARMY  RSCH  LABORATORY  -  HRED 
ATTN  AMSRD  ARL  HR  MV  HQ  USAOTC 
S  MIDDLEBROOKS 
91012  STATION  AVE  ROOM  111 
FT  HOOD  TX  76544-5073 

1  ARMY  RSCH  LABORATORY  -  HRED 
ATTN  AMSRD  ARL  HR  MY  M  BARNES 
2520  HEALY  AVE  STE  1 172  BLDG  51005 
FT  HUACHUCA  AZ  85613-7069 
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ARMY  RSCH  LABORATORY  -  HRED 
ATTN  AMSRD  ARL  HR  MP  D  UNGV ARSKY 
BATTLE  CMD  BATTLE  LAB 
415  SHERMAN  AVE  UNIT  3 
FT  LEAVENWORTH  KS  66027-2326 

ARMY  RSCH  LABORATORY  -  HRED 
ATTN  AMSRD  ARL  HR  M  DR  B  KNAPP 
ARMY  G1  MANPRINT  DAPE  MR 
300  ARMY  PENTAGON  ROOM  2C489 
WASHINGTON  DC  20310-0300 

ARMY  RSCH  LABORATORY  -  HRED 
ATTN  AMSRD  ARL  HR  MJK  MS  D  BARNETTE 
JFCOM  JOINT  EXPERIMENTATION  J9 
JOINT  FUTURES  LAB 
115  LAKEVIEW  PARKWAY  SUITE  B 
SUFFOLK  VA  23435 

ARMY  RSCH  LABORATORY  -  HRED 
ATTN  AMSRD  ARL  HR  MQ  M  R  FLETCHER 
US  ARMY  SBCCOM  NATICK  SOLDIER  CTR 
AMSRD  NSC  SSE  BLDG  3  RM  341 
NATICK  MA  01760-5020 

ARMY  RSCH  LABORATORY  -  HRED 
ATTN  AMSRD  ARL  HR  MT  DR  J  CHEN 
12350  RESEARCH  PARKWAY 
ORLANDO  FL  32826-3276 

ARMY  RSCH  LABORATORY  -  HRED 
ATTN  AMSRD  ARL  HR  MS  MR  C  MANASCO 
SIGNAL  TOWERS  RM  3  03 A 
FORT  GORDON  GA  30905-5233 

ARMY  RSCH  LABORATORY  -  HRED 
ATTN  AMSRD  ARL  HR  MU  M  SINGAPORE 
6501  E  11  MILE  RD  MAIL  STOP  284 
BLDG  200A  2ND  FL  RM  2104 
WARREN  MI  48397-5000 

ARMY  RSCH  LABORATORY  -  HRED 
ATTN  AMSRD  ARL  HR  MF  MR  C  HERNANDEZ 
BLDG  3040  RM  220 
FORT  SILL  OK  73503-5600 

ARMY  RSCH  LABORATORY  -  HRED 
ATTN  AMSRD  ARL  HR  MW  E  REDDEN 
BLDG  4  ROOM  332 
FT  BENN1NG  GA  31905-5400 


1  ARMY  RSCH  LABORATORY  -  HRED 
ATTN  AMSRD  ARL  HR  MN  R  SPENCER 
DCSFDI  HF 

HQ  USASOC  BLDG  E2929 
FORT  BRAGG  NC  28310-5000 

1  ARL-HRED  LIAISON 

PHYSICAL  SCIENCES  LAB 
PO  BOX  30002 

LAS  CRUCES  NM  88003-8002 
1  DIRECTOR 

UNIT  OF  ACTION  MANEUVER  BATTLE  LAB 
ATTN  ATZKUA 
BLDG  1101 

FORT  KNOX  KY  40121 
ABERDEEN  PROVING  GROUND 
1  DIRECTOR 

US  ARMY  RSCH  LABORATORY 
ATTN  AMSRD  ARL  Cl  OK  (TECH  LIB) 

BLDG  4600 

1  DIRECTOR 

US  ARMY  RSCH  LABORATORY 
ATTN  AMSRD  ARL  Cl  OK  S  FOPPIANO 
BLDG  459 

1  DIRECTOR 

US  ARMY  RSCH  LABORATORY 

ATTN  AMSRD  ARL  HR  MR  F  PARAGALLO 

BLDG  459 
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